Imaging device, imaging device image output method, and computer program

ABSTRACT

An image pickup apparatus capable of performing dynamic assignment of an output image in a dynamic range according to a subject.  
     The image pickup apparatus includes an image pickup device ( 131 ) for picking up an image of a subject, a signal processing section ( 133 ) for generating a composite image having a relatively wider dynamic range than at least either of the dynamic ranges of a long-time exposure image picked up with a relatively long exposure time by the image pickup device and a short-time exposure image picked up with a relatively short exposure time by the image pickup device, by synthesizing the long-time exposure image and the short-time exposure image, and a control section ( 137 ) for compressing the composite image and dynamically varies the assignment proportion of a high luminance dynamic range to a low-middle luminance dynamic range in a dynamic range of an output image to be outputted as a video signal.

TECHNICAL FIELD

The present invention relates to an image pickup apparatus and, moreparticularly, to an image pickup apparatus having an expandable dynamicrange.

BACKGROUND ART

Image pickup apparatuses such as video cameras, digital video cameras orstill video cameras having image pickup devices such as CCDs (ChargeCouple Devices) are widely and generally used.

Image pickup devices such as CCDs are narrow in dynamic range comparedto silver halide cameras. For this reason, when an image is takenagainst backlight with such an image pickup device, lost highlightdetail which causes a bright section to lose grayscale or lost shadowdetail which causes a dark section to lose grayscale occur in areproduced image.

In a conventional type of image pickup apparatus, the amount of exposureis adjusted to become correct for a subject, by an automatic exposurefunction, but there still remain many cases where although a mainsection of a subject has correct exposure, lost highlight detail and thelike occurs in a background and the like.

In addition, in the conventional type of image pickup apparatus, backcompensation has been performed in such a manner that thereproducibility of an image at a low luminance signal level/a middleluminance signal level (a low-middle luminance signal level) is weightedwhile the detail of a high luminance section is lost. However, there arealso cases where lost highlight details in a background desire to bemade visible.

For this reason, in order to cope with an image of wide dynamic rangecontaining a bright section and a dark section as mentioned above, therehas been provided an image pickup apparatus (wide dynamic range camera)capable of taking an image of wide dynamic range by synthesizing animage whose main subject is a comparatively bright section captured by ahigh speed shutter or the like and an image whose main subject is acomparatively dark section captured by a low speed shutter or the like.

However, in the above-mentioned image pickup apparatus capable of takingan image of wide dynamic range, such as a wide dynamic range camera, theproportion in which the dynamic range of an output image reproducible asa video signal is assigned to a high luminance section in which theluminance signal levels correspond to high luminance and to a low-middleluminance section in which the luminance signal levels correspond to lowluminance/middle luminance is constantly fixed irrespective of the kindof subject.

Accordingly, for example, if an image signal of the high luminancesection is absent in the output image, an image signal assigned to thehigh luminance section in the dynamic range of the output image isabsent, while the remaining low-middle luminance section only isassigned to the dynamic range. This hinders effective use of the dynamicrange and provides a generally dark image.

The present invention has been conceived in view of the above-mentionedproblems, and an object of the present invention is to provide a noveland improved image pickup apparatus capable of performing dynamicassignment in the dynamic range of an output image according to the kindof subject.

DISCLOSURE OF THE INVENTION

To solve the above-mentioned problems, according to a first aspect ofthe present invention, an image pickup apparatus may include an imagepickup device for picking up an image of a subject, a signal processingsection for generating a composite image having a relatively widerdynamic range than at least either of the dynamic ranges of a long-timeexposure image picked up with a relatively long exposure time by theimage pickup device and a short-time exposure image picked up with arelatively short exposure time by the image pickup device, bysynthesizing the long-time exposure image and the short-time exposureimage, and a control section for compressing the composite image anddynamically varies the proportion of a high luminance dynamic range to alow-middle luminance dynamic range in a dynamic range of an output imageto be outputted as a video signal.

According to the present invention, a long-time exposure image and ashort-time exposure image are synthesized on the basis of an imagesignal picked up by the image pickup device, whereby a composite imageof expanded dynamic range is generated. Furthermore, the proportion inwhich the dynamic range of the composite image is assigned to either thehigh luminance dynamic range or the low-middle luminance dynamic rangeof an output image is dynamically varied to compress and assign the highluminance dynamic range and the low-middle luminance dynamic range ofthe composite image. In this construction, for example, even if apicked-up image is a dark image in which a signal amount occupying itshigh luminance region is small as a whole, the proportion is dynamicallyvaried to an appropriate assignment proportion, whereby clear images maybe constantly outputted with correct grayscale. The assignmentproportion means the proportion in which the dynamic range is assignedto either the high luminance dynamic range or the low-middle luminancedynamic range.

The control section dynamically varies the assignment proportion of thehigh luminance dynamic range to the low-middle luminance dynamic rangeaccording to at least a luminance region which occupies the compositeimage. In this construction, the assignment proportion may be variedaccording to a variation in the degree of brightness which occupies thecomposite image. Accordingly, for example, even if a high luminanceregion in the luminance region decreases and a dark image occurs,correct grayscale may be obtained by increasing the proportion ofassignment to the low-middle luminance dynamic range with respect to theproportion of assignment to the high luminance dynamic range.

The control section corrects the assignment proportion of the highluminance dynamic range to the low-middle luminance dynamic range eachtime the composite image is generated. In this construction, each timethe composite image is generated, it is determined whether theassignment proportion of the dynamic ranges is appropriate, and ifinappropriate, the assignment proportion is corrected. Accordingly, itis possible to constantly output an image reproduced with appropriategrayscale.

The luminance region is at least either a high luminance region or alow-middle luminance region. In this construction, it is possible torealize assignment of the dynamic ranges by taking note of the highluminance region, the low-middle luminance region or the high luminanceregion the low-middle luminance region. In addition, the luminanceregion is not limited to the above-mentioned one, and may also be madeof, for example, a high luminance region, a middle luminance region anda low luminance region.

The control section dynamically varies the assignment proportion of thehigh luminance dynamic range to the low-middle luminance dynamic rangeaccording to at least an average luminance signal level of the highluminance region which occupies the composite image. In thisconstruction, the assignment proportion is dynamically varied accordingto variations in a bright section (the high luminance region) of thecomposite image. Accordingly, it is possible to weight the low-middleluminance region which is contained in many subjects, and increase thelow-middle luminance dynamic range.

The control section may be adapted to dynamically vary the assignmentproportion of the high luminance dynamic range to the low-middleluminance dynamic range according to at least an average luminancesignal level of the low-middle luminance region which occupies thecomposite image.

The control section dynamically varies the assignment proportion of thehigh luminance dynamic range to the low-middle luminance dynamic rangeaccording to at least the high luminance region which occupies thecomposite image. In this construction, the assignment proportion isdynamically varied according to variations in a bright section (the highluminance region) of the composite image. For example, if a brightsection decreases, the grayscale of a dark section is increased so thatthe reproducibility thereof may be improved.

The control section may be adapted to dynamically vary the assignmentproportion of the high luminance dynamic range to the low-middleluminance dynamic range according to at least the low-middle luminanceregion which occupies the composite image.

The control section at least monotonically varies the assignmentproportion of the high luminance dynamic range to the low-middleluminance dynamic range. This construction makes it possible togradually vary the assignment proportion. Accordingly, since a sharpvariation does not occur, the variation of the assignment proportion maybe applied in many kinds of locations. The present invention is notlimited to the above-mentioned example, and may also be carried out inthe case where the assignment proportion is varied like a quadraticcurve.

The luminance region may be configured to be at either a high luminanceregion or a low-middle luminance region.

Among pixels constructed in the composite image, pixels corresponding toat least a higher luminance signal level than a switch luminance signallevel may be acquired from the short-time exposure image, while pixelscorresponding to a lower luminance signal level than the switchluminance signal level may be acquired from the long-time exposureimage.

Among pixels constructed in the composite image, pixels corresponding toat least a higher luminance signal level than a switch luminance signallevel may be acquired from the long-time exposure image, while pixelscorresponding to a lower luminance signal level than the switchluminance signal level may be acquired from the short-time exposureimage.

The dynamic range is at least either a high luminance dynamic range or alow-middle luminance dynamic range.

The present invention is characterized in that a compression gain forcompressing a luminance signal level of the composite image isdetermined on the basis of at least the assignment proportion of a highluminance dynamic range of an output image to a low-middle luminancedynamic range thereof.

The present invention may be constructed to determine a high luminancecompression gain for compressing a luminance signal level of the highluminance region in the composite image and a low-middle luminancecompression gain for compressing a luminance signal level of thelow-middle luminance region in the composite image.

The present invention may be constructed to determine for each luminancesignal level of the composite image at least either a final highluminance compression gain or a final low-middle luminance compressiongain which is to be used for compression, on the basis of at leasteither the high luminance compression gain or the low-middle luminancecompression gain.

The present invention may be constructed to dynamically vary theassignment proportion of the high luminance dynamic range to thelow-middle luminance dynamic range of the output image according to atleast the high luminance region which occupies the composite image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view showing a schematic construction of a widedynamic range camera according to the present embodiment.

FIG. 2 is an explanatory view showing a schematic subject in a case inwhich the interior a house having a window according to the presentembodiment.

FIG. 3 is a block diagram showing a schematic construction of the widedynamic range camera according to the present embodiment.

FIG. 4 is a block diagram showing a schematic construction of a signalpreprocessing block according to the present embodiment.

FIGS. 5A to 5C are explanatory views schematically showing thesynchronization processing of a timing adjustment section according tothe present embodiment.

FIG. 6 is a flowchart schematically showing the image output processingof an image pickup apparatus according to the present embodiment.

FIG. 7 is an explanatory view schematically showing the input/outputcharacteristics of images during synthesis processing according to thepresent embodiment.

FIG. 8 is a cumulative histogram schematically showing a distribution ofthe luminance signal levels of a long-time exposure image according tothe present embodiment.

FIG. 9 is a cumulative histogram schematically showing a distribution ofthe luminance signal levels of a short-time exposure image according tothe present embodiment.

FIG. 10 is an explanatory view schematically showing the input/outputcharacteristic of an output image according to the present embodiment.

FIG. 11 is an explanatory view schematically showing the input/outputcharacteristic of an output image according to the present embodiment.

FIG. 12 is an explanatory view schematically showing a correctionfunction based on the area of a high luminance region according to thepresent embodiment.

FIG. 13 is an explanatory view schematically showing a correctionfunction based on an average luminance signal level of a high luminanceregion according to the present embodiment.

BEST MODES FOR CARRYING OUT THE INVENTION

A preferred embodiment of the present invention will be described belowin detail with reference to the accompanying drawings. In the followingdescription and accompanying drawings, constituent elements havingapproximately identical functions are denoted by identical referencenumerals, and repetition of the same description is omitted herein.

(1. Wide Dynamic Range Camera)

First, a wide dynamic range camera (wide D camera) according to thepresent embodiment will be described with reference to FIGS. 1 and 2.FIG. 1 is a perspective view showing a schematic construction of thewide dynamic range camera according to the present embodiment. FIG. 2 isan explanatory view showing a schematic subject in a house which has awindow and whose indoor state is being captured according to the presentembodiment. An image pickup apparatus 102 according to the presentembodiment is capable of picking up a still image and/or a moving image.

As shown in FIG. 2, when the image pickup apparatus which is generallyused as a digital video camera, a still video camera and the like is topick up an indoor image of a house having a window 122 in, for example,a daytime period in fine weather, if the image pickup apparatus sets anexposure reference for a subject to the person 120 who is presentindoors, the window 122 which is brighter than the person 120 losesgrayscales and suffers loss of highlight detail.

This is because the image pickup apparatus may not handle a dynamicrange which extends over a wide range from the luminance of the person120 which is relatively smaller than that of the window 122 to theluminance of the window 122 which is relatively brighter than that ofthe person 120.

To solve the above problem, there is an image pickup apparatus called awide dynamic range camera (wide D camera). The image pickup apparatushas, for example, the function of performing processing such as varyingthe shutter speed of an electronic shutter and synthesizing a pluralityof images each having a different exposure time, so that even if anexposure reference for a subject is set to the person 120 as shown inFIG. 2, the window 122 may be prevented from suffering loss of highlightdetail and reproduced as a good grayscale image.

As techniques for picking up an image of a subject having a wide dynamicrange to be reproduced as an output image containing various sectionsfrom a bright section to a dark section, there are, in addition to theabove-mentioned method of synthesizing a bright image and a dark imageeach having a different exposure time, a method of changing thesensitivity of an image pickup device in units of pixels, extractingonly signals having the same exposure condition from the image pickupdevice, reproducing images, and synthesizing one or not less than twoimages each having a different exposure condition, and a method ofdividing incident light by means of a prism and synthesizing a signaloutputted from an image pickup device and a signal outputted from animage pickup device to which is stuck a device having a light reductionfunction like an ND filter (Neutral Density Filter; a light amountadjustment filter) which reduces transmitted light overall wavelengths,i.e., equally reduces the amount of incident light.

The wide D camera 102 shown in FIG. 1 has a dynamic range far wider thanthose of generally used video cameras, and is capable of picking up animage of a subject having a wide dynamic range to be reproduced as anoutput image containing various sections from a bright section to a darksection. Accordingly, the wide D camera 102 is suitable for picking upan image at an indoor location which is shone with strong externallight, a location exposed to greatly different light intensities, andthe like.

For example, the wide D camera 102 is particularly used as asurveillance camera or the like which often picks up images in the casewhere the dynamic range greatly varies between time periods when imagepickup is to be performed, such as the daytime and the nighttime. Inaddition, the wide D camera 102 used in the present embodiment is notlimited to the surveillance camera.

(2. Construction of Each Component of Wide Dynamic Range Camera)

The construction of each component of the wide D camera 102 according tothe present embodiment will be described below with reference to FIG. 3.FIG. 3 is a block diagram showing a schematic construction of the widedynamic range camera according to the present embodiment. In FIG. 3, forexample, a microcomputer 137 corresponds to a control section of thepresent invention, but is not limited to this example.

As shown in FIG. 3, the wide D camera 102 according to the presentembodiment may include an optical component 130, an image pickup device131, a preprocessing section 132, a signal processing section 133, anoutput section 134, a TG 135, an EVR 136, the microcomputer 137(hereinafter referred to as the microcomputer 137), an optical-componentcontrol circuit 138, and a recording medium 139.

The optical component 130 is an optical filter which reduces unnecessarywavelengths, such as an ND filter. Light incident from a subject ontothe optical component 130 and transmitted therethrough is picked up bythe image pickup device 131, and is converted into an electrical signal.In addition, the optical component 130 has, for example, a lens iris andthe like to be controlled by the microcomputer 137 via theoptical-component control circuit 138. The microcomputer 137 will bedescribed later in detail.

The image pickup device (image pickup device) 131 is capable ofphotoelectrically converting an optical image received from the subjectand outputting an electrical image signal, by means of a plurality ofpixels respectively made of photoelectric conversion devices disposedtwo-dimensionally on a light receiving surface. For example, the imagepickup device 131 is a solid-state image pickup device such as a CCDmade of a multiplicity of kinds.

The preprocessing section 132, if the image pickup device 131 is a CCD,receives the above-mentioned image signal outputted from the imagepickup device 131, and then takes out a video signal while reducingnoise (noise) by the processing of CDS (a correlated double samplingcircuit; correlated double sampling).

Furthermore, the preprocessing section 132 gives a gain to the videosignal to amplify the video signal to a suitable signal level asoccasion demands, and transmits the video signal to the signalprocessing section 133.

The main noise of the CCD which constitutes the image pickup device 131may include, for example, transfer noise, noise of an output amplifier,reset noise, noise caused by dark current, and light shot noise.

Among the above-mentioned kinds of noise, the transfer noise is noiseproduced when charges are being transferred, but rarely becomes aproblem since buried channel CCDs are generally used. The noise of theoutput amplifier is noise of a source follower, and if this noise isreduced, the noise of the CCD is ameliorated.

The reset noise is noise produced when the FD (floating diffusion) ofthe CCD is reset, and CDS makes it possible to reduce this reset noise.

The signal processing section 133 is made of two major blocks. One is asignal preprocessing block 140 which performs extraction of an inputtedlong-time exposure image and an inputted short-time exposure image eachhaving a different exposure time, correction of the color balancebetween the long-time exposure image and the short-time exposure image,and synthesis and compression of the long-time exposure image and theshort-time exposure image. The exposure times of the long-time exposureimage and the short-time exposure image may be individually setaccording to subjects whose images are to be picked up. The signalpreprocessing block 140 will be described later in detail.

The other block following the signal preprocessing block is a signalpost procesing block (not shown) which performs color processing such asaperture (aperture) processing for performing γ(gamma) correction andcontour correction, and white balance for adjusting color temperature onthe basis of “white” contained in the subject.

Furthermore, the signal processing section 133 has at least a detectionsection 143 and is capable of detecting level information such as aluminance signal level, color information or the like of an image signalinputted at the present point in time. The signal processing section 133is capable of calculating gains such as compression gains and adjustingwhite balance on the basis of the detected information. The detectionsection 143 will be described later in detail.

The output section 134 receives the video signal from the signalprocessing section 133 after the processing of the signal processingsection 133, and outputs a video image to a monitor such as a displaydevice through a driver of the output section 134 in order to reproducethe video image.

The TG (timing generator) 135 generates pulses necessary for the imagepickup device 131 made of a CCD or the like. For example, pulses such as4-phase pulses for vertical transfer, field shift pulses, and 2-phasepulses for horizontal transfer are generated. The image pickup device131maybe driven (electronic shutter function) by the TG 135.

By adjusting the driving of the image pickup device 131, it is possibleto pick up a long-time exposure image having a relatively long exposuretime and a short-time exposure image having a relatively short exposuretime. In the image pickup apparatus 102 which is the wide dynamic rangecamera according to the present embodiment, image pickup processing isperformed by the above-mentioned processing, but image pickup processingaccording to the present embodiment is not limited to this example.

The EVR (electronic volume) 136 is a variable resistor whose resistancevalue may be adjusted by digital signals or the like, and controls theoptical-component control circuit 138 and the like by varying theresistance value. Accordingly, iris adjustment of a lens whichconstitutes the optical component 130 may be performed by the EVR 136and the like. In addition, the EVR 136 has storage means such as amemory, and may hold a varied resistance value when a power source isturned off.

The optical-component control circuit 138 has control means (not shown)for performing control such as iris adjustment of the optical component130 or switching from one optical filter to another such as an ND filteramong a plurality of optical components 130.

The microcomputer 137 controls the processing of each of thepreprocessing section 132, the signal processing section 133, the TG135, the EVR 136 and the optical-component control circuit 138 on thebasis of, for example, a detection result from the above-mentioneddetection section 143. In addition, the microcomputer 137 determinescompression gains (compression gains) and the like for synthesizing andcompressing a long-time exposure image (long) and a short-time exposureimage (short) of the image pickup apparatus 102 which is the widedynamic range camera. The microcomputer 137 will be described later indetail.

Furthermore, the microcomputer 137 may control the image pickupapparatus 102 by mutually communicating, for example, control data forcontrolling the apparatus with an apparatus outside of the image pickupapparatus 102 via an “external I/F (external interface)”. The “externalI/F” is, for example, Ethernet (registered trademark) such as 10Base-Tor 10Base-2, EIA-232, EIA-485 or the like.

The storage medium 139 is capable of storing, for example, control datanecessary for control of individual sections provided in the imagepickup apparatus 102, adjustment data for adjustment of exposure ofsubjects, user setting data such as color correction and AE which may bevariously set by individual users who use the image pickup apparatus102.

The storage medium 139 may use, for example, a ROM which may store datawritten thereto, and an EEPROM (Electrically Erasble and ProgrammableROM) which may store or erase data by electrical erasure.

(2.1 Signal Preprocessing Block 140)

The signal preprocessing block included in the signal processing section133 according to the present embodiment will be described below withreference to FIG. 4. FIG. 4 is a block diagram showing a schematicconstruction of the signal preprocessing block according to the presentembodiment.

In FIG. 4, for example, the microcomputer 137 corresponds to the controlsection of the present invention, while a compression gain calculationsection 145 corresponds to a compression gain calculation section of thepresent invention, but this example is not restrictive. Furthermore,although the compression gain calculation section 145 is provided in thesignal preprocessing block 140, the present invention is not limited tothis example, and may also be carried out even in various other casessuch as the case where the compression gain calculation section 145 isprovided in the microcomputer 137 as the control section, and the casewhere the compression gain calculation section 145 is provided in asection other than the signal preprocessing block 140.

As shown in FIG. 4, the signal preprocessing block 140 may include atiming adjustment section 141, clamp processings 142, a detectionsection 143, a synthesis section 144, the compression gain calculationsection 145, and a compression section 146.

The timing adjustment section 141 adjusts (synchronizes) the timings ofimage signals having different exposure times, which are respectivelyoutputted from the preprocessing section 132 at different timings, to asynchronized timing. Synchronization processing in the timing adjustmentsection 141 will be described below.

First, as shown in FIG. 5A, a long-time exposure image signal (longsignal) containing a long-time exposure image (long) picked up by theimage pickup device 131, and a short-time exposure image signal (shortsignal) containing a short-time exposure image (short) picked up by thesame are transmitted from the preprocessing section 132 to the timingadjustment section 141 via one bus (path) shown in FIG. 4 in such amanner that the long signal and the short signal are alternatelyarranged by time sharing. FIGS. 5A to 5C are explanatory viewsschematically showing the synchronization processing of the timingadjustment section 141 according to the present embodiment.

When the long-time exposure image signal and the short-time exposureimage signal are alternately inputted to the timing adjustment section141 in a time-shared manner, the timing adjustment section 141 performssynchronization on the long-time exposure image and the signalshort-time exposure image signal.

The above-mentioned synchronization extracts the long-time exposureimage and the signal short-time exposure image signal contained in onesignal, as shown in FIG. 5A, and adjusts (performs synchronization on)the temporal timing of the long-time exposure image contained in thelong-time exposure image signal and the temporal timing of theshort-time exposure image contained in the short-time exposure imagesignal, as shown in FIG. 5B (long-time exposure image signal) and 5C(short-time exposure image signal). As shown in FIG. 4, the timingadjustment section 141 separately outputs the short-time exposure imageand the long-time exposure image to the respective clamp processingsections 142.

Since the synchronization is performed in the timing adjustment section141, switching of pixels may be smoothly performed when the long-timeexposure image and the short-time exposure image are to be synthesizedin the synthesis section 144, whereby the synthesis section 144 mayefficiently carry out synthesis processing.

Then, the clamp processing sections 142 determine a “0 level (blacklevel)” reference from the luminance signal levels of the long-timeexposure image and the short-time exposure image (clamp processing).After the determination of the 0 level, each of the clamp processingsections 142 outputs the corresponding one of the long-time exposureimage and the short-time exposure image to both the synthesis section144 and the detection section 143.

The respective clamp processing sections 142 perform reproduction ofdirect current components, reduce low frequency noises, and determinewhere the 0 level is, by making use of the fact that the long-timeexposure image signal and the short-time exposure image signal haveperiodicity. In composite video signals and luminance signals, blacklevels are used as references, and direct current values representinformation. Accordingly, in signal processing, the black levels arefixed, and the processing is performed on the basis of these levels.

The detection section 143 detects what signal amount is inputted andwhat the luminance signal level is, as to each of the long-time exposureimage signal and the short-time exposure image signal outputted from theclamp processing sections 142. The luminance signal level (luminancelevel) indicates the degree of brightness.

Namely, when a detection result of the detection section 143 isoutputted to individual sections such as the microcomputer 137, theindividual sections become able to execute processing. The detectionsection 143 detects the signal amount or the distribution of theluminance signal level (a histogram of the luminance signal level) ofeach of the inputted long-time and short-time exposure images so thatthe microcomputer 137 in particular may determine a switch point forsynthesis of the long-time exposure image and the short-time exposureimage.

The synthesis section 144 performs level adjustment on the exposurelevels of the inputted long-time exposure image and short-time exposureimage, and synthesizes both exposure images into one image by performingswitch processing in units of pixels. The generated composite image isoutputted to the compression section 146 and the compression gaincalculation section 145 as a composite image signal. The synthesisprocessing of the synthesis section 144 to synthesize the long-timeexposure image and the short-time exposure image will be described laterin detail.

The microcomputer 137 may receive the signal amounts or the histogramsof the luminance signal levels detected from the respective long-timeand short-time exposure image signals. In addition, an average luminancesignal level and the like of a high luminance region or a low-middleluminance region may be found from the signal amount of each of theluminance signal levels.

The microcomputer 137, on the basis of information such as the detectionresult acquired from the detection section 143, performs processing suchas adjustment of the exposures of the short-time exposure image and thelong-time exposure image, correction of the proportion of a highluminance dynamic range and a low-middle luminance dynamic range in thedynamic range of an output image to be outputted as a video signal,determination of the switch point for synthesis of the short-timeexposure image and the short-time exposure image, or determination ofthe compression gains (compression gains) of the generated compositeimage. The above-mentioned switch point or compression gains and thelike are transmitted to each of the processing sections of the signalpreprocessing block 140 by the microcomputer 137. In addition, therespective compression gains are constructed for different luminanceregions. The compression gains in the present embodiment include a highluminance compression gain (Gs) and a low-middle luminance compressiongain (Gl), but this example is not restrictive. For example, thecompression gains may be made of a high luminance compression gain, alow-middle luminance compression gain and a low luminance compressiongain. The operation processing of the microcomputer 137, such asdetermination of the switch point, correction of the proportion of thedynamic ranges and determination of the compression gains will bedescribed later in detail.

During compression of the generated composite image, the compressiongain calculation section (Compression Gain Calculator) 145 calculatesfinal compression gains for the respective luminance signal levels to becompressed, on the basis of the above-mentioned compression gains (thehigh luminance compression gain (Gs) and the low-middle luminancecompression gain (Gl)) transmitted from the microcomputer 137, andtransmits the final compression gains to the compression section 146.The final compression gains in the present embodiment include a finalhigh luminance compression gain and a final low-middle luminancecompression gain, but this example is not restrictive.

The compression section 146 compresses the dynamic range of thecomposite image synthesized by the synthesis section 144 to the dynamicrange of an output image to be outputted as a video signal, on the basisof the inputted final compression gains. The compressed output image istransmitted to the next signal postprocessing block (not shown).

The output image outputted from the compression section 146 is inputtedto the signal postprocessing block, in which color processing such asgamma correction, aperture or white balance is executed.

In the color processing, since the gamma characteristic of, for example,a CRT (Cathode Ray Tube, a picture tube) is determined, the image pickupapparatus 102 must perform correction in advance so that a reproducedpicture may obtain correct grayscale characteristics. Accordingly, thecorrection needs to be performed by gamma correction. In general, diodecharacteristics are used.

(3. Image Output Operation of Image Pickup Apparatus 102)

An embodiment of the image output operation of the image pickupapparatus 102 constructed in the above-mentioned manner will bedescribed below.

(3.1 Image Processing (S160) of Image Pickup Apparatus)

First, the image pickup processing of the image pickup apparatus 102will be described below with reference to FIGS. 3, 4 and 6. FIG. 6 is aflowchart schematically showing the image output processing of the imagepickup apparatus according to the present embodiment.

First, as shown in FIG. 6, the image pickup apparatus 102 which is awide dynamic camera performs image pickup processing (S160) of a subjectby means of the image pickup device 131. Although the image pickupprocessing according to the present embodiment is image pickupprocessing based on a double exposure method, the present invention isnot limited to this example, and may also be carried out in the case of,for example, a 2-CCD type which uses two image pickup devices 131 suchas CCDs and has a standard-speed shutter and a high-speed shutter.

The double exposure method is a method of picking up a long-timeexposure image of a subject which needs a relatively long exposure timeand whose correct exposure is in a low-middle luminance region and ashort-time exposure image of a subject whose correct exposure is in ahigh luminance region, and synthesizing both images to expand anapparent dynamic range. The term “luminance region” means a regionformed by an aggregation of pixels or signals constructed in an imagehaving approximately the same luminance level. For example, a highluminance region denotes a region formed by pixels or signals of highluminance levels, among pixels or signals constructed in an image.

After the above-mentioned image pickup processing (S160), the long-timeexposure image (long) and the short-time exposure image (short) whichhave been outputted from the image pickup device 131 and subjected to,for example, CDS processing by the preprocessing section 132 aretransmitted to the timing adjustment section 141 of the signalpreprocessing block 140 of the signal processing section 133 shown inFIG. 4.

The above-mentioned both image signals are alternately inputted to thetiming adjustment section 141 via one bus at different timings in such amanner that the long-time exposure image (long) and the short-timeexposure image (short) are time-shared, so that both image signals needto be synchronized as described above.

(3.2. Detection Processing (S162) of Image Pickup Apparatus 102)

After the long-time exposure image and the short-time exposure imagehave been synchronized in the timing adjustment section 141, bothexposure images are separately outputted from the timing adjustmentsection 141 and inputted to the clamp processing sections 142. Afterhaving been subjected to clamp-processing in the clamp processingsections 142, the long-time exposure image and the short-time exposureimage are transmitted to each of the detection section 143 and thesynthesis section 144.

When the long-time exposure image and the short-time exposure image aretransmitted to the synthesis section 144, the synthesis section 144performs detection processing (S162) on the long-time exposure image andthe short-time exposure image.

The synthesis section 144 detects through the detection processing(S162) of the long-time exposure image and the short-time exposureimage, for example, signal amounts at the respective signal levels ofthe long-time exposure image, signal amounts at the respective signallevels of the short-time exposure image, a distribution (histogram) ofthe signal amounts at the respective signal levels of the long-timeexposure image, or a distribution of the signal amounts at therespective signal levels of the short-time exposure image.

On the basis of the detection result in the detection processing (S162),the microcomputer 137 determines La (switch luminance signal level)indicative of the highest luminance signal level in the low-middleluminance region in the long-time exposure image, and Lb indicative ofthe highest luminance signal level in the high luminance region in theshort-time exposure image. La and Lb will be described later in detail.

(3.3. Synthesis Processing (S164) of Image Pickup Apparatus 102)

After the detection processing (S162), the synthesis processing (S164)is performed on the long-time exposure image and the short-time exposureimage in the synthesis section 144, whereby a composite image isgenerated. The synthesis section 144 performs the synthesis processing(S164) on the basis of information transmitted from the microcomputer137, for example, La. The synthesis processing will be described belowin detail.

The synthesis processing according to the present embodiment performsswitching on the basis of the switch point (Switch point) so as to adoptpixels corresponding to the luminance signal levels of the long-timeexposure image as to luminance signal levels lower than the switch pointand, when the switch point is exceeded, adopt pixels corresponding tothe luminance signal levels of the short-time exposure image.

Accordingly, one composite image is synthesized by switching a target tobe adopted, from the long-time exposure image to the short-time exposureimage in units of pixels. The switch point according to the presentembodiment will be described below with reference to FIG. 7. FIG. 7 isan explanatory view schematically showing the input/outputcharacteristics of images during the synthesis processing according tothe present embodiment.

As shown in FIG. 7, the input/output characteristic of the short-timeexposure image (short) is a short-time input/output characteristic 170,the input/output characteristic of the long-time exposure image (long)is a long-time input/output characteristic 171, and the input/outputcharacteristic of the output image outputted from the signal processingsection 133 as the video signal is an output image input/outputcharacteristic 172. The horizontal axis shown in FIG. 7 represents theluminance signal levels of image signals inputted to the synthesissection 144, while the vertical axis represents the luminance signallevels of image signals outputted from the synthesis section 144.

First, for synthesis, the exposure ratio of the long-time exposure imageto the short-time exposure image is multiplied by the short-timeexposure image, whereby the levels of both images are adjusted. Forexample, if the exposure ratio of the long-time exposure image to theshort-time exposure image is 10:1, the exposure of the short-timeexposure image is one-tenth of the long-time exposure image. However,the amount of existing light is ten times the luminance signal levels ofthe short-time exposure image. Accordingly, the levels are adjusted bymultiplying the short-time exposure image by 10.

Accordingly, the inclination of the short-time input/outputcharacteristic 170 moves in the direction of the arrow shown in FIG. 7,and the level of the short-time input/output characteristic 170coincides with that of the long-time input/output characteristic 171.Furthermore, on the basis of an appropriate switch point (switch point),the short-time input/output characteristic 170 is inclined by the amountof predetermined inclination, whereby the output image input/outputcharacteristic 172 is obtained.

The predetermined inclination is stored in, for example, the storagemedium 139, and further, the microcomputer 137 performs the processingof inclining the short-time input/output characteristic 171level-adjusted by the above-mentioned multiplication, by the amount ofthe predetermined inclination. The reason for the short-timeinput/output characteristic 171 is inclined by the above-mentionedamount of inclination is that the dynamic range is very wide and noisesuch as image distortion needs to be avoided.

When the level adjustment of the above-mentioned both images iscompleted, pixels to be adopted for a composite image are switched fromthe long-time exposure image to the short-time exposure image on thebasis of the switch point (switch point) shown in FIG. 7, whereby onecomposite image is synthesized. Accordingly, a composite image isgenerated which has the characteristics of the long-time exposure imagein which a dark section corresponding to the low-middle luminance signallevels is reproduced with good grayscale, and the characteristics of theshort-time exposure image in which a bright section corresponding to thehigh luminance signal levels is reproduced with good grayscale, wherebythe bright section and the dark section are reproduced with goodgrayscale.

(3.3.1 Exclusion Processing of Synthesis Section 144)

In the above-mentioned synthesis processing (S164), the pixels for thecomposite image are not equally adopted from the luminance signal levelsof both the long-time exposure image and the short-time exposure image,and if there is not a signal amount corresponding to a luminance signallevel, the exclusion processing of excluding the luminance signal levelfrom the target to be synthesized is performed.

This means that the luminance signal level is not assigned to (isexcluded from) the dynamic range of the composite image. Accordingly, itis possible to make effective use of the dynamic range to be assigned tothe composite image.

The exclusion processing according to the present embodiment for makingeffective use of the dynamic range in the synthesis processing (S164)will be described below with reference to FIGS. 8 and 9. FIG. 8 is acumulative histogram schematically showing a distribution of theluminance signal levels of the long-time exposure image according to thepresent embodiment, and FIG. 9 is a cumulative histogram schematicallyshowing a distribution of the luminance signal levels of the short-timeexposure image according to the present embodiment.

First, as shown in FIG. 8, as the luminance signal level becomesgradually higher toward the highest luminance signal level La (switchluminance signal level) in the low-middle luminance region, the signalamount is accumulated. When the luminance signal level exceeds La, thesignal amount is not accumulated in the range of luminance signal levelsshown as a range 180. Accordingly, in the range 180, a signal or a pixelis absent. In FIGS. 8 and 9, the horizontal axis represents theluminance signal levels of image signals inputted to the synthesissection 144, while the vertical axis represents a cumulative signalamount.

If a signal or a pixel corresponding to a luminance signal level isabsent like in the range 180 shown in FIG. 8, the luminance signal levelcorresponding to the range 180 is excluded from the target for thesynthesis processing (S164). Accordingly, the excluded luminance signallevel is not assigned to the dynamic range of the composite image,whereby it is possible to make effective use of the dynamic range.

Furthermore, although signals or pixels corresponding to luminancesignal levels in a range 182, shown in FIG. 8, which is a high luminanceregion, are present, the luminance signal levels overlap the luminancesignal levels of the short-time exposure image shown in FIG. 9 whichwill be mentioned later. Since the luminance signal levels of the highluminance region in the long-time exposure image in particular overlapthe luminance signal levels of the short-time exposure image, theluminance signal levels are excluded from the target for the synthesisprocessing (S164) Although the synthesis processing according to thepresent embodiment has been described with illustrative reference to thecase where the overlapping luminance signal levels are excluded, thepresent invention is not limited to this example, and may also becarried out in the case where, for example, a luminance signal level isset in advance and excluded.

Namely, as shown in FIG. 8, the pixels corresponding to the luminancesignal levels in the long-time exposure image for the synthesisprocessing (S164) from which the range 180 and the range 182 areexcluded become a target to be adopted for the composite image.Accordingly, the range of luminance signal levels to be adopted for thesynthesis processing (S164) is a range lower than La. Namely, by settingthe above-mentioned La as the switch point (switch point) shown in FIG.7, it is possible to efficiently use necessary pixels the long-timeexposure image, and it is possible to make effective use of the dynamicrange of the composite image. Although La in the present embodiment isdetermined by the microcomputer 137, the present invention is notlimited to this example, and may also be carried out in the case of, forexample, the detection section 143.

The luminance regions according to the present embodiment are regionseach having a predetermined luminance signal level range in an image,and the region in which luminances relatively correspond to low-middleluminances is made the low-middle luminance region, while the region inwhich luminances relatively correspond to high luminances is made thehigh luminance region. However, the present invention is not limited tothis example, and may also be carried out in the case where, forexample, the low-middle luminance region is further divided into a lowluminance region and a middle luminance region.

Then, as shown in FIG. 9, the highest luminance signal level of the highluminance region in the short-time exposure image is set to Lb, andpixels corresponding to lower luminance signal levels than Lb are made atarget to be applied to the composite image.

As described previously, in the short-time exposure image as well, if asignal or a pixel corresponding to a luminance signal level is absent,the luminance signal level is excluded from the target for the synthesisprocessing (S164). Accordingly, the excluded luminance signal level isnot assigned to the dynamic range of the composite image, whereby it ispossible to make effective use of the dynamic range with high useefficiency by assigning a section in which grayscales are absent toother luminance signal levels.

Furthermore, the present embodiment has been described with illustrativereference to the case where the switch point (switch point) is La(switch luminance signal level), but the present invention is notlimited to this example and may also be carried out in the case where aluminance signal level which is set in advance is used as a switchpoint.

(3.4 Compression Processing (S168) in Image Pickup Apparatus 102)

In the image output operation of the image pickup apparatus 102according to the present embodiment, the next processing following thesynthesis processing (S164) is assignment processing (S166) as shown inFIG. 6, but compression processing (S168) will first be described below.

The dynamic range of the composite image synthesized in the synthesisprocessing (S164) is far wider than that of an image of, for example, agenerally used type of video camera.

Accordingly, none of the processing sections (devices) provided in theimage pickup apparatus 102 which is the wide dynamic range camera isable to process the composite image signal, so that the dynamic range ofthe composite image needs to be compressed to a processable dynamicrange (the dynamic range of the output image).

First, the compression processing according to the present embodimentwill be described below with reference to FIG. 10. FIG. 10 is anexplanatory view schematically showing the input/output characteristicof the output image according to the present embodiment. In FIG. 10, thehorizontal axis represents the luminance signal level of an image signalinputted to the compression section 146, while the vertical axisrepresents the luminance signal level of an image signal outputted fromthe compression section 146.

As shown in FIG. 10, the input/output characteristic of the compositeimage is a composite input/output characteristic 202, and theinput/output characteristic of the output image in which the dynamicrange of the composite image is compressed is the output imageinput/output characteristic 172. The output image is outputted from thesignal preprocessing block 140 as a video signal, and is transmitted tothe next signal postprocessing block (not shown).

The proportion of a low-middle luminance dynamic range to a highluminance dynamic range in the dynamic range of the composite imageshown in FIG. 10 is a:b, where b=1−a. The proportion of a low-middleluminance dynamic range to a high luminance dynamic range in the dynamicrange of the output image is m:n, where n=1−m.

The dynamic range (dynamic range) according to the present embodimentmay include the high luminance dynamic range which is the dynamic rangeof the high luminance region and the low-middle luminance dynamic rangewhich is the dynamic range of the low-middle luminance region.

The dynamic range is the range of brightness of a subject which may behandled by an image pickup device such as an image pickup device, i.e.,the range of reproducible grayscales.

Accordingly, the brightness or grayscale of an image dramatically variesaccording to what proportion of the grayscale of a subject should beassigned to which of the high luminance dynamic range and the low-middleluminance dynamic range in the above-mentioned dynamic range. Forexample, if the grayscale of the high luminance region is to be weightedin the dynamic range of the output image, the proportion of n in thereproduction ratio of m:n is increased to increase the proportion of thehigh luminance dynamic range, whereby the detail of the grayscale of thehigh luminance region is reproduced.

Then, the compression gains for compressing the dynamic range of thecomposite image are found by the microcomputer 137. The compressiongains found by the microcomputer 137 are the low-middle luminancecompression gain (Gl) found on the basis of La in the low-middleluminance region and the high luminance compression gain (Gs) found onthe basis of Lb in the high luminance region.

Since the low-middle luminance dynamic range of the composite image atLa is compressed to the low-middle luminance dynamic range of the outputimage as shown in FIG. 10, the low-middle luminance compression gain(Gl) may be expressed by the following equation:Gl=m/La   (Eq. 1)

La or Lb is found by the microcomputer 137 on the basis of the detectionresult by the detection section 143, such as the signal amounts or thehistogram of the luminance signal levels.

Similarly, the high luminance compression gain is found by themicrocomputer 137. Since the range of the high luminance regioncontinues to Lb, the high luminance region between Lb and La needs to beaccommodated into n. Therefore, the high luminance compression gain (Gs)at Lb may be expressed by the following equation:Gs=n/(Lb−La)   (Eq. 2)

The high luminance dynamic range and the low-middle luminance dynamicrange of the composite image which are found by the equations (1) and(2) are respectively compressed by the compression section 146 in thedirections indicated by the arrows shown in FIG. 10, and the dynamicrange of the composite image is assigned to the dynamic range of theoutput image.

Namely, it may be said that the assignment proportion of the highluminance dynamic range to the low-middle luminance dynamic range in thedynamic range of the composite image is assigned to the assignmentproportion of the high luminance dynamic range to the low-middleluminance dynamic range in the dynamic range of the output image.

In addition, the low-middle luminance compression gain (Gl) and the highluminance compression gain (Gs) which are respectively found by theequations (1) and (2) are compression gains at the input luminancesignal levels La and Lb, respectively. Accordingly, to be exact, thecompression gains at the individual luminance signal levels between Laand Lb differ from Gl or Gs.

The final compression gains are found by the compression gaincalculation section 145. The compression gain calculation section 145finds, on the basis of Gl and Gs transmitted from the microcomputer 137,a final low-middle luminance compression gain at each luminance signallevel in the low-middle luminance region and a final high luminancecompression gain at each luminance signal level in the high luminanceregion. Then, the compression gain calculation section 145 transmits thefinal high luminance compression gain and the final low-middle luminancecompression gain to the compression section 146.

(3.5 Assignment Processing (S166) of Image Pickup Apparatus 102)

The assignment processing (S166) according to the present embodimentwill be described below with reference to FIG. 11. FIG. 11 is anexplanatory view schematically showing the input/output characteristicof the output image according to the present embodiment.

Approximately like in FIG. 10, the input/output characteristic of theoutput image shown in FIG. 11 is the output image input/outputcharacteristic 172, the input/output characteristic of a correctedoutput image is a corrected output image input/output characteristic173, and the input/output characteristic of the composite image is thecomposite input/output characteristic 202. The proportion (reproductionratio) of the low-middle luminance dynamic range to the high luminancedynamic range in the dynamic range of the output image is m:n, wheren=1−m. In FIG. 11, the horizontal axis represents the luminance signallevel of an image signal inputted to the compression section 146, whilethe vertical axis represents the luminance signal level of an imagesignal outputted from the compression section 146.

For example, when no signals at all exist in the high luminance regionof a composite image, if an output image is reproduced by assigning thedynamic range of the composite image to only the range “m” which is thelow-middle luminance dynamic range of the low-middle luminance region inthe dynamic range of the output image, the output image exhibits agenerally dark grayscale. The above-mentioned assignment of the dynamicrange may be deemed inappropriate assignment and inefficient assignment.

Accordingly, the microcomputer 137 determines whether the proportion ofa high luminance dynamic range to a low-middle luminance dynamic rangein the dynamic range of an output image outputted previously isappropriate for a composite image to be newly inputted, and if theproportion is inappropriate, the microcomputer 137 performs correctionby changing the proportion of the high luminance dynamic range to thelow-middle luminance dynamic range to an appropriate proportion.Accordingly, it is possible to achieve efficient and appropriateassignment of the dynamic range.

To enable the above-mentioned assignment, the assignment proportion ofthe high luminance dynamic range to the low-middle luminance dynamicrange in the dynamic range of the output image is dynamically varied foreach input composite image. The dynamic change of the proportionaccording to the present embodiment is executed each time a compositeimage is inputted, but the present invention is not limited to thisexample and may also be carried out in the case where the assignmentproportion of the dynamic ranges is changed each time, for example, fivecomposite images are inputted.

In the dynamic change of the assignment proportion of the high luminancedynamic range to the low-middle luminance dynamic range, the proportionof the high luminance dynamic range to the low-middle luminance dynamicrange, i.e., m:n, is corrected according to the signal amount of thehigh luminance region which occupies the composite image.

The correction of the proportion of the high luminance dynamic range tothe low-middle luminance dynamic range according to the presentembodiment is performed on the basis of the proportion of the highluminance region found from the signal amount of the high luminanceregion (the area of the high luminance region) or the average luminancesignal level of the high luminance region. The case of the area of thehigh luminance region and the case of the average luminance signal levelof the high luminance region will be separately described below.

The correction of the dynamic range according to the present embodimentwill be described with illustrative reference to the case where thecorrection is performed according to the high luminance region occupyingthe composite image, but the present invention is not limited to thisexample and may also be carried out in the case where the dynamic rangeis corrected on the basis of an area or an average luminance signallevel found from the signal amount of the low luminance region orlow-middle luminance region.

(3.5.1 Assignment Processing (S166) Based on Area of High LuminanceRegion)

If m′:n′ is used to denote a corrected proportion obtained by correctingthe proportion (m:n) of the low-middle luminance dynamic range to thehigh luminance dynamic range, m′ and n′ may be expressed by thefollowing equations:m′=f(x)×n+m   (Eq. 3)n′=1−m′  (Eq. 4)

In the equation (3), f(x) is a correction function based on the area ofthe high luminance region. The input/output characteristic of thecorrection function f(x) will be described below with reference to FIG.12. FIG. 12 is an explanatory view schematically showing the correctionfunction based on the area of the high luminance region.

As shown in FIG. 12, the input/output characteristic of the correctionfunction is an area input/output characteristic 220. The horizontal axisrepresents the proportion of the area of the high luminance region inthe composite image, while the vertical axis represents correctionvalues of the correction function f(x).

As shown in FIG. 12, when the proportion of the area of the highluminance region is less than Rb and not less than Ra, the correctionvalue of the correction function f (x) takes any value from “1.0” to“0”. Furthermore, m′ and n′ of the corrected proportion are found fromthe equations (3) and (4), whereby the assignment processing (S166) isperformed. If the proportion of the area is not less than Ra, the entirehigh luminance dynamic range is assigned to the low-middle luminancedynamic range in the assignment processing (S166), while if theproportion of the area exceeds Rb, the assignment processing (S166) isnot performed and the proportion of the high luminance dynamic range tothe low-middle luminance dynamic range remains unchanged.

Accordingly, as shown in FIGS. 11 and 12, if the proportion of the areaof the high luminance region is less than Rb and not less than Ra, asthe proportion of the area of the high luminance region approaches Ra,m′ of the corrected proportion increases. Namely, the corrected outputimage input/output characteristic 173 moves on the La axis in thevertically upward direction. This fact means that since the highluminance region decreases, the proportion in which the high luminancedynamic range is assigned to the low-middle luminance dynamic rangeincreases. In contrast, as the proportion of the area of the highluminance region approaches Rb, m′ of the corrected proportiondecreases, and the corrected output image input/output characteristicshown in FIG. 11 moves on the La axis in the vertically downwarddirection, and approaches m.

In addition, as shown in FIG. 12, if the proportion of the area of thehigh luminance region is less than Ra, all the high luminance dynamicrange of the output image assigned to the high luminance region isassigned to the low-middle luminance dynamic range.

Accordingly, as the high luminance region decreases, the proportion ofthe dynamic range to be assigned to the low-middle luminance dynamicrange increases, whereby the grayscale reproducibility of the low-middleluminance region is improved. Namely, the proportion of the highluminance dynamic range to the low-middle luminance dynamic range isdynamically varied for each inputted output image according to thesignal amount of the high luminance region thereof, whereby the dynamicrange is corrected into a dynamic range appropriate for the outputimage.

Although the above description has referred to the example in which thecorrection function f(x) according to the present embodiment draws amonotonous variation, the present invention is not limited to thisexample and may also be carried out in the case where the correctionfunction f(x) draws, for example, a quadratic curve.

The assignment processing (S166) based on the average luminance signallevel of the high luminance region has approximately the sameconfiguration as the assignment processing (S166) based on the area ofthe high luminance region, and as shown in FIG. 13, the input/outputcharacteristic based on the average luminance signal level of the highluminance region assumes an average luminance signal level 230, and thehorizontal axis represents the average luminance signal level of thehigh luminance region.

Accordingly, as shown in FIGS. 11 and 13, if the average luminancesignal level of the high luminance region is less than Rb and not lessthan Ra, as the proportion of the area of the high luminance regionapproaches Ra, m′ of the corrected proportion increases. Namely, thecorrected output image input/output characteristic 173 moves on the Laaxis in the vertically upward direction. This fact means that since thehigh luminance region decreases, the proportion in which the highluminance dynamic range is assigned to the low-middle luminance dynamicrange increases.

In addition, as shown in FIG. 13, if the average luminance signal levelof the high luminance region is less than Ra, all the high luminancedynamic range of the output image assigned to the high luminance regionis assigned to the low-middle luminance dynamic range.

Accordingly, as the signal amount of the high luminance regiondecreases, the proportion of the dynamic range to be assigned to thelow-middle luminance dynamic range increases, whereby the grayscalereproducibility of the low-middle luminance region is improved. Namely,the proportion of the high luminance dynamic range to the low-middleluminance dynamic range is dynamically varied for each inputted outputimage according to the signal amount of the high luminance regionthereof, whereby the dynamic range is corrected into a dynamic rangeappropriate for the output image. Incidentally, m′ and n′ of thecorrected proportion are found by the microcomputer 137.

In addition, Ra and Rb according to the present embodiment may beappropriately varied according to the characteristics of the imagepickup apparatus 102, the kind of subject whose image is to be pickedup, and the like.

Furthermore, the present invention may also be carried out in the casewhere both the area of the high luminance region and the averageluminance signal level thereof are used in combination. In this case, ifx and x′ are respectively used to denote the proportion of the area ofthe high luminance region and the average luminance signal level of thehigh luminance section, the correction function is expressed as F(x,x′). In addition, if the corrected proportion is made m′:n′ like in theabove description, m′ and n′ may be expressed by the followingequations:m′=F(x, x′)×n+m   (Eq. 5)n′=1−m′  (Eq. 6)

Since the equations (5) and (6) have approximately the sameconfiguration as the above-described ones, the detailed description ofthe equations (5) and (6) is omitted.

When the compression processing (S168) is completed, the compressionsection 146 transmits the output image to the signal postprocessingblock as a video signal, and one sequence of image output operations iscompleted.

The dynamic range according to the present embodiment may include thelow-middle luminance dynamic range which is the dynamic range of aregion having luminances relatively corresponding to low-middleluminances, and the high luminance dynamic range which is the dynamicrange of a region having luminances relatively corresponding to highluminances. However, the present invention is not limited to thisexample and may also be carried out in the case where, for example, thelow-middle luminance dynamic range is further divided into a lowluminance dynamic range and a middle luminance dynamic range.

Although the preferred embodiment of the present invention has beendescribed above with reference to the accompanying drawings, the presentinvention is not limited to this example. It will be readily apparent tothose skilled in the art that various changes or modifications may bemade without departing from the technical idea set forth in the appendedclaims, and all such changes and modifications are, of course,considered to fall within the technical scope of the invention.

Although the above description of the embodiment has referred to theexample in which the image pickup apparatus 102 picks up ablack-and-white image, the present invention is not limited to thisexample, and may also be carried out in the case where the image pickupapparatus 102 picks up, for example, a color image or both a color imageand a black-and-white image.

INDUSTRIAL APPLICABILITY

As described hereinabove, according to the present invention, dynamicranges respectively assigned to a long-time exposure image and ashort-time exposure image are dynamically varied according to the areaof a high luminance region or the average luminance signal level of ahigh luminance section, whereby images may be reproduced with, forexample, grayscale and exposure constantly maintained under optimumconditions. Furthermore, during synthesis, an unnecessary section suchas the range of luminance signal levels in which signals are absent orthe range of luminance signal levels overlapping the short-time exposureimage is excluded from the long-time exposure image, whereby it ispossible to make far more efficient use of the dynamic ranges.

1. An image pickup apparatus characterized by comprising: an imagepickup device for picking up an image of a subject; a signal processingsection for generating a composite image having a relatively widerdynamic range than at least either the dynamic ranges of a long-timeexposure image picked up with a relatively long exposure time by saidimage pickup device or a short-time exposure image picked up with arelatively short exposure time by said image pickup device, bysynthesizing said long-time exposure image and said short-time exposureimage; and a control section for compressing said composite image anddynamically varying the assignment proportion of a high luminancedynamic range to a low-middle luminance dynamic range in a dynamic rangeof an output image to be outputted as a video signal.
 2. An image pickupapparatus according to claim 1, characterized in that said controlsection dynamically varies said assignment proportion of said highluminance dynamic range to said low-middle luminance dynamic rangeaccording to at least a luminance region which occupies said compositeimage.
 3. An image pickup apparatus according to claim 1, characterizedin that said control section corrects said assignment proportion of saidhigh luminance dynamic range to said low-middle luminance dynamic rangeeach time said composite image is generated.
 4. An image pickupapparatus according to claim 3, characterized in that said luminanceregion is at least either a high luminance region or a low-middleluminance region.
 5. An image pickup apparatus according to claim 1,characterized in that said control section dynamically varies saidassignment proportion of said high luminance dynamic range to saidlow-middle luminance dynamic range according to at least an averageluminance signal level of said high luminance region which occupies saidcomposite image.
 6. An image pickup apparatus according to claim 1,characterized in that said control section dynamically varies saidassignment proportion of said high luminance dynamic range to saidlow-middle luminance dynamic range according to at least an averageluminance signal level of said low-middle luminance region whichoccupies said composite image.
 7. An image pickup apparatus according toclaim 1, characterized in that said control section dynamically variessaid assignment proportion of said high luminance dynamic range to saidlow-middle luminance dynamic range according to at least said highluminance region which occupies said composite image.
 8. An image pickupapparatus according to claim 1, characterized in that said controlsection dynamically varies said assignment proportion of said highluminance dynamic range to said low-middle luminance dynamic rangeaccording to at least said low-middle luminance region which occupiessaid composite image.
 9. An image pickup apparatus according to claim 8,characterized in that said control section at least monotonically variessaid assignment proportion of said high luminance dynamic range to saidlow-middle luminance dynamic range.
 10. An image pickup apparatuscharacterized by comprising: an image pickup device for picking up animage of a subject; a signal processing section for generating acomposite image having a relatively wider dynamic range than the dynamicrange of at least either a long-time exposure image picked up with arelatively long exposure time by said image pickup device or ashort-time exposure image picked up with a relatively short exposuretime by said image pickup device, by synthesizing said long-timeexposure image and said short-time exposure image; and a control sectionfor compressing said composite image and dynamically assigning thedynamic range of said composite image to the dynamic range of an outputimage to be outputted as a video signal.
 11. An image pickup apparatusaccording to claim 10, characterized in that said control sectiondynamically assigns the dynamic range of said composite image to thedynamic range of said output image according to at least a luminanceregion which occupies said composite image.
 12. An image pickupapparatus according to claim 10, characterized in that said controlsection dynamically assigns the dynamic range of said composite image tothe dynamic range of said output image each time said composite image isgenerated.
 13. An image pickup apparatus according to claim 10,characterized in that said control section dynamically assigns thedynamic range of said composite image to the dynamic range of saidoutput image according to at least a high luminance region whichoccupies said composite image.
 14. An image pickup apparatus accordingto claim 10, characterized in that said control section dynamicallyassigns the dynamic range of said composite image to the dynamic rangeof said output image according to at least an average luminance signallevel of said high luminance region which occupies said composite image.15. An image pickup apparatus according to claim 10, characterized inthat said dynamic range is at least either a high luminance dynamicrange or a low-middle luminance dynamic range.
 16. An image pickupapparatus according to claim 15, characterized in that said controlsection dynamically assigns assignment of at least either said highluminance dynamic range or said low-middle luminance dynamic range ofsaid output image according to said high luminance region which occupiessaid composite image.
 17. An image pickup apparatus according to claim15, characterized in that said control section dynamically assignsassignment of at least either said high luminance dynamic range or saidlow-middle luminance dynamic range of said output image according tosaid high luminance region which occupies said composite image.
 18. Animage pickup apparatus according to claim 15, characterized in that saidcontrol section dynamically assigns a section of said high luminancedynamic range of said output image to said low-middle luminance dynamicrange according to at least a decrease of said high luminance regionwhich occupies said composite image.
 19. An image pickup apparatusaccording to claim 15, characterized in that said control sectiondynamically assigns a section of said high luminance dynamic range ofsaid output image to said low-middle luminance dynamic range accordingto at least a decrease of an average luminance signal level of said highluminance region which occupies said composite image.
 20. An imagepickup apparatus according to claim 15, characterized in that saidcontrol section dynamically assigns a section of said low-middleluminance dynamic range of said output image to said high luminancedynamic range according to at least an increase of said high luminanceregion which occupies said composite image.
 21. An image pickupapparatus according to claim 15, characterized in that said controlsection dynamically assigns a section of said low-middle luminancedynamic range of said output image to said high luminance dynamic rangeaccording to at least an increase of an average luminance signal levelof said high luminance region which occupies said composite image. 22.An image pickup apparatus according to claim 21, characterized in thatsaid control section at least monotonically varies the assignmentproportion of said high luminance dynamic range to said low-middleluminance dynamic range of the said output image.
 23. An image pickupapparatus characterized by comprising: an image pickup device forpicking up an image of a subject; a detection section for detecting animage signal of a long-time exposure image picked up with a relativelylong exposure time by said image pickup device, and an image signal of ashort-time exposure image picked up with a relatively short exposuretime by said image pickup device; a synthesis section for generating acomposite image from said long-time exposure image and said short-timeexposure image on the basis of a switch luminance signal leveldetermined from said image signals; a control section for compressingsaid composite image according to a luminance region which occupies saidcomposite image, and dynamically assigning the dynamic range of anoutput image to be outputted as a video signal; and a compressionsection for compressing the dynamic range of said composite image on thebasis of dynamic assignment of said dynamic range of said output image.24. An image pickup apparatus according to claim 23, characterized inthat said luminance region is at either a high luminance region or alow-middle luminance region.
 25. An image pickup apparatus according toclaim 23, characterized in that said synthesis section acquires, fromsaid short-time exposure image, said pixels corresponding to at least ahigher luminance signal level than said switch luminance signal levelamong pixels constructed in said composite image.
 26. An image pickupapparatus according to claim 23 characterized in that said synthesissection acquires, from said long-time exposure image, said pixelscorresponding to at least a lower luminance signal level than saidswitch luminance signal level among said pixels constructed in saidcomposite image.
 27. An image pickup apparatus according to claim 23,characterized in that said dynamic range is at least either a highluminance dynamic range or a low-middle luminance dynamic range.
 28. Animage pickup apparatus according to claim 23, characterized in that saidcontrol section determines a compression gain for compressing aluminance signal level of said composite image on the basis of at leastthe assignment proportion of a high luminance dynamic range of saidoutput image to a low-middle luminance dynamic range thereof.
 29. Animage pickup apparatus according to claim 23, characterized in that saidcontrol section determines at least a high luminance compression gainfor compressing a luminance signal level of said high luminance regionin said composite image and a low-middle luminance compression gain forcompressing a luminance signal level of said low-middle luminanceregion.
 30. An image pickup apparatus according to claim 23,characterized in that said control section further includes acompression gain calculation section for determining, for each luminancesignal level of said composite image, at least either a final highluminance compression gain or a final low-middle luminance compressiongain which are to be used by said compression section, on the basis ofat least either said high luminance compression gain or said low-middleluminance compression gain.
 31. An image pickup apparatus according toclaim 23, characterized in that said control section dynamically variesan assignment proportion between said high luminance dynamic range andsaid low-middle luminance dynamic range of said output image accordingto at least said high luminance region which occupies said compositeimage.
 32. An image pickup apparatus according to claim 31,characterized in that said control section at least monotonically variesthe assignment proportion of said high luminance dynamic range to saidlow-middle luminance dynamic range of said output image.
 33. An imagepickup apparatus characterized by comprising: an image pickup device forpicking up an image of a subject; a detection section for detecting animage signal of a long-time exposure image picked up with a relativelylong exposure time by said image pickup device, and an image signal of ashort-time exposure image picked up with a relatively short exposuretime by said image pickup device; a synthesis section for excluding saidluminance signal level from a target to be synthesized, when at leastsaid image signal corresponding to a luminance signal level of eithersaid long-time exposure image or said short-time exposure image isabsent, and synthesizing said long-time exposure image and saidshort-time exposure image on the basis of said switch luminance signallevel; a control section which dynamically assigns the dynamic range ofan output image to be outputted as a video signal in which a compositeimage is compressed, according to a luminance region which occupies saidcomposite image in which said long-time exposure image and saidshort-time exposure image are synthesized; and a compression section forcompressing the dynamic range of said composite image on the basis ofdynamic assignment of said dynamic range of said output image.
 34. Animage pickup apparatus according to claim 33, characterized in that saidsynthesis section selects said luminance signal level lower than saidswitch luminance signal level in said long-time exposure image, as atarget for said composite image.
 35. An image pickup apparatus accordingto claim 33, characterized in that said synthesis section selects saidluminance signal level higher than said switch luminance signal level insaid short-time exposure image, as a target for said composite image.36. An image pickup apparatus according to claim 33, characterized inthat said luminance region is at least either a high luminance region ora low-middle luminance region.
 37. An image pickup apparatus accordingto claim 33, characterized in that said dynamic range is at least eithera high luminance dynamic range or a low-middle luminance dynamic range.38. An image pickup apparatus according to claim 33, characterized inthat said control section dynamically varies the assignment proportionof said high luminance dynamic range to said low-middle luminancedynamic range of said output image according to at least a highluminance region which occupies said composite image.